System and Method for Tracking a Deformation

ABSTRACT

An imaging system to reconstruct a reflectivity image of a scene including an object moving with the scene. A tracking system to track a deforming object to estimate an object deformation for each time step. Sensors acquire snapshots of the scene, each acquired snapshot of the object includes measurements in the object deformation for that time step, to produce a set of object measurements with deformed shapes over the time steps. Compute a correction to estimates of object deformation for each time step, with matching measurements of the corrected object deformation for each time step to measurements in the acquired snapshot of object for that time step. Select a corrected deformation over other corrected deformations for each time step, according to a distance between the corrected deformation and the estimate of the deformation, to obtain a final estimate of the deformation of the deformable object moving in the scene.

FIELD

The present disclosure relates generally to sensing systems, and moreparticularly to sensing of deformable objects moving in a scene.

BACKGROUND

In several remote sensing applications, acquiring high-resolution radarimages is necessary in order to meet some of the application andbusiness requirements. For example, radar reflectivity imaging is usedin various security, medical, and through-the-wall imaging (TWI)applications. Whereas the down-range resolution is mostly controlled bythe bandwidth of the transmitted pulse, the cross-range (azimuth)resolution depends on the aperture of the radar sensors. Typically, thelarger the aperture, the higher the image resolution is, regardless ofwhether the aperture is physical (a large antenna) or synthetic (amoving antenna). Currently, the increase of the physical size of antennaleads to a significant increase of the cost of the radar system. To thatend, a number of radar imaging systems use synthetic-aperture methods toreduce the size of the antennas and the cost of radar imaging. Forexample, synthetic-aperture radar (SAR) and inverse SAR (ISAR) use therelative motion of the radar antenna and an object in the scene toprovide finer spatial resolution with comparatively small physicalantennas, i.e., smaller than the antennas of beam-scanning radars.

However, the small size of physical antennas of radar systems makes thetracking of deformable moving objects difficult. Specifically, trackingobjects exhibiting arbitrary motion and deformation requires trackingsensitivity with minimum resolution greater than the resolution of thephysical antennas resulting in an impractical cost of the radar imagingsystem. To that end, conventional radar and/or other electromagnetic oracoustic wave imaging systems require the object to be standing still inthe scene or moving in a very controlled rigid motion. Even for therigid motion, conventional radar imaging systems require a challengingtracking step to estimate the motion parameters of the moving objectusing only the radar data, before a radar image can be formed, see,e.g., Martorella 2014. (Martorella, M. (2014). Introduction to inversesynthetic aperture radar. In Academic Press Library in Signal Processing(Vol. 2, pp. 987-1042). Elsevier.)

Therefore, there is a need for imaging systems and methods suitable fordetermining unknown deformations or other permutations that might affecta signal during acquisition, or correct errors of an estimateddeformation of a signal.

SUMMARY

The present disclosure relates to acquisition systems including sensingsystems acquiring a signal under an unknown permutation(s), such assensing of deformable objects moving in a scene.

Some embodiments relate to imaging systems, and more particularly toimaging systems that image a deformable object moving or undergoingdeformations as it is being acquired using one or more snapshots. Inthese embodiments, the imaging system may reconstruct the image of theobject under one or more deformations and may represent the object in aprototypical deformation.

In some embodiments the imaging system may comprise of one or more ofthe following sensors: camera, depth camera, radar, magnetic resonanceimaging (MRI), ultrasonic, computer assisted tomography (CAT), LIDAR,terahertz, and hyperspectral, among others. One or more of those sensorsmay be used for tracking the deformation and one or more may be used forimaging the object. In some embodiments, the same sensor, or sensors,might be used to achieve both tracking and imaging.

Some embodiments provide an imaging system, for example comprising of anoptical camera and a depth sensor, that allows tracking the motion of anobject even if the object is deformable and the motion is not rigid.Some embodiments further provide a second imaging system, such as aradar or an ultrasonic array, that images the object as the object movesand deforms within the scene. Wherein the second imaging systemreconstructs the image of the object moving in the scene with aresolution greater than a resolution governed by practically sizedphysical sensors, such as arrays of electromagnetic or ultrasonicsensors acquiring reflectivity images.

Some embodiments provide a radar imaging system suitable for airportsecurity applications allowing a person to freely move in front of radarimaging system, while the radar imaging system is reconstructing a radarreflectivity image of the person. Some types of sensors used ingathering image data include using optical sensors, such as monochromeor color or infrared video cameras, or depth cameras or a combinationthereof. The optical sensors are less expensive than electromagneticsensors, along with operating in a modality that provides easiertracking of a target. Hence, an optical sensor can be used for trackingthe motion of the target, even if the target is deformable and themotion is not rigid.

Further, some embodiments are based on another recognition that in anumber of applications where radar imaging of deformable objects isnecessary and useful, the object moving sufficiently close and visibleto the radar imaging system, such that optical sensors can providesufficient accuracy for tracking. Wherein, some embodiments are based onrealization that by aiding the radar reconstruction using the opticalmotion tracking, the radar imaging system can be able to image verycomplex target objects that are moving.

An example where the target is clearly visible is security applications,in which people walk in front of a scanning system, e.g., in an airport.In some airport security scanners require subjects to be standing in aspecific pose to be scanned for prohibited items. The scanning systemaccording to one embodiment allows the subjects (which are thedeformable moving objects, such as humans) to simply walk through thescanner while they are scanned, without any need to stop.

Some embodiments of the present disclosure include a radar imagingsystem configured to determine a radar reflectivity image of a sceneincluding an object moving with the scene. The radar imaging systemincludes an optical sensor to track the object over a period of time toproduce, for each time step, an object deformation. The radar imagingsystem can also include one or more electromagnetic sensors, such as ammWave sensor, a THz imaging sensor, or a backscatter X-Ray sensor, orcombinations thereof, to acquire snapshots of the object over themultiple time steps. Each snapshot includes measurements representing aradar reflectivity image of the object with a deformed shape defined bythe corresponding deformation. Wherein, what was recognized is that oneof the reasons preventing electromagnetic sensors of a radar imagingsystem to track a moving object, is a resolution of the electromagneticsensing governed by a physical size of the antennas of the sensors.Specifically, for the practical reasons, the size of the antennas of theradar imaging system can allow to estimate only coarse image of theobject at each time step. Such a coarse image can be suitable to trackan object subject to rigid and finite transformation but can fail torecognize arbitrarily non-rigid transformation typical for the motion ofa human.

Other embodiments of the present disclosure are based on anotherrecognition that a radar imaging system can jointly use measurements ofa scene acquired over multiple time steps. Such a system of measurementscan be used to improve the resolution of the radar reflectivity imagebeyond a resolution governed by the size of the antennas of the radarimaging system. However, when the object is moving over time, atdifferent time steps, the object can be located at different positionsand can have a different shape caused by the non-rigid motion. Such adislocation and deformation of the object make the system ofmeasurements ambiguous, i.e., ill-posed, and difficult or impractical tosolve. In particular, not rigidly moving objects can have differentshape at different instances of time. To that end, at different timesteps there can be different deformations of the shape of the objectwith respect to its nominal shape and different transformations of aradar reflectivity image observed by the radar imaging system withrespect to a radar reflectivity image of the object.

Other embodiments of the present disclosure are based on anotherrecognition that an imaging system might be mounted on a mounted on amoving platform, obtaining snapshots of its surroundings as it moves,and that deformation of its input is due to changes in the geometry ofthe environment as the imaging system moves with the platform. Thus,each snapshot of the environment includes a deformation, and thedeformation itself provides information about the motion of the sensorand the moving platform in the environment. In addition, rough, or moreprecise determination of the deformation can often be performed usingone of many approaches in the art, known collectively as simultaneouslocalization and mapping (SLAM) methods. Embodiments of the presentdisclosure can be used to refine the output of completely replace SLAMmethods.

For example, some embodiments of the present disclosure use an existingSLAM algorithm in the art, to compute an estimate of the deformation ofthe scene that is observed by the sensors. This estimate is refined suchthat the data acquired by the sensors at each snapshot is matched whenthe refined deformation estimate is applied.

In many problems in the art, including SLAM, unlabeled sensing, andimaging of deformable objects while in motion, there is a problem ofrecovering a signal that is measures subject to unknown perturbation.Some embodiments of the present disclosure are based on the realizationthat in most practical permutations, the unknown permutations are notarbitrary but some unknown permutations are more likely to occur thanothers.

Based on this realization, and to further exploit this, some embodimentsof the present disclosure include a regularization function thatpromotes the more likely permutations in the solution. Throughexperimentation, what was learned from this approach is that, eventhough the general problem is not convex, an appropriate relaxation ofthe resulting regularized problem allowed for an exploiting of thewell-developed machinery of the theory of optimal transport (OT), and todevelop a tractable algorithm.

A key realization that allows using OT to develop a tractable algorithmis that an unknown deformation or an unknown permutation of one signalto another is equivalent to transporting notional mass between pixelssuch that the mass transported from one signal to the other is inducinga deformation of the signal. The theory of OT can therefore guide thistransport such that it happens optimally, i.e., recovers the optimaldeformation or permutation that explains the acquired snapshots.

A further realization is that the existence of a notion of a transportcost in the OT theory can be used to provide regularization that favorsmore likely permutations or deformation. In particular, the theory ofoptimal transport (OT) determines a mass transport plan that is optimalwhen considering the total cost of transferring the mass, wherein thecost of transferring the mass from one pixel to another can bedetermined by the application. If a deformation of the signal is morelikely than another, then the corresponding total cost of the transportof each pixel in this deformation is lower than the corresponding costin a less likely deformation and, thus, the transport corresponding tomore likely deformation is preferable by OT recovery theory andalgorithms.

Another realization is that in some practical applications thedeformations and permutations that are most likely are the ones in whichpixels are not transported very far from their original location and,therefore, the cost of moving the pixel to a nearby location is lowerthan the cost of moving the pixel to a farther location. Therefore, atransport cost that penalizes mass moving closer less than mass movingfarther can be used as a regularization. Since such transport costs arewell studied in the art of OT, this realization provides the use of muchbetter developed OT algorithms to estimate the OT plan.

A similar realization is that some other practical applications thedeformations and permutations that are most likely are the ones in whichpixels are not transported very far from where their nearby pixels aretransported and, therefore, the cost of moving the pixel to a newlocation is lower if the nearby pixels are also moved nearby the newlocation, compared to the cost of moving the pixel to location fartherfrom where the nearby pixels are moved to. Therefore, a transport costthat penalizes mass moving together less than mass separating can beused as a regularization. Since such transport costs are also wellstudied in the art of optimal transport (OT), this realization providesthe use of much better developed OT algorithms to estimate the OT plan.

Another key realization is that certain deformations might includeocclusions of parts of the signal, and different snapshots might exhibitdifferent deformations that include different occlusions of the signal.Furthermore, certain patterns of occlusion are more likely to occur thanothers. For example, since nearby pixels of an object move together,they are more likely to also be occluded together from another part ofthe object. As an example, a human walking in front of a camera mightswing the arms as part of the walking motion. It is very likely that thewhole arm away from the camera is occluded by the body. Furthermore,nearby points on the arm are likely to be occluded together as the armmoves behind the body. The closer the points are the more likely theyare to be behind the body at the same time.

In these cases, optimal transport (OT) theory allows for additional costto be considered in the total cost when adding or removing mass from thesignal. In the OT art, the subfield is sometimes referred to asunbalanced OT or partial OT. However, existing methods in the art do notconsider that mass, i.e., pixels, located nearby is more likely todisappear or appear together than mass not located together. For thisreason, some embodiments of the present disclosure may introduce adifferent cost in computing the plan that incorporates the structure ofthe mass difference between the two deformations in order to reduce thecost of deformations in which nearby pixels appear or disappeartogether, and thus consider such deformations more likely than one inwhich the pixels appearing and disappearing are not nearby.

Some embodiments of the present disclosure include systems and methodsto determine signals that have been observed with multiple snapshots,subject to a different permutation in each snapshot. Wherein, thesesystems and methods exploit the knowledge that certain permutations aremore likely than others, in order to effectively determine the signal.Such that, by using optimal transport theory to incorporate thisknowledge in the solution, these systems and methods can determine theunknown signal much more effectively than the conventional imagingsystem approaches.

For example, some test approaches included using an alternative modalityto track the deformable object. Other test approaches included animaging system for multimodal imaging with deformations, assuming onemodality is used to determine the deformation and another modality toperform the imaging. Wherein these approaches taught that whendetermining the deformation, the deformation introduced errors, and thatthese test approaches provided some simple methods to correct theseinduced errors. Unfortunately, what was later discovered after furthertest experiments is that these test approaches simply did do not workvery well. For example, the modality used to track the object did nothave the resolution required by the radar system and the tracking wasprone to errors. In such approaches, the reconstruction was notaccurate. It is, therefore, desirable that the imaging process shouldalso refine the tracking and correct the imprecisions in estimating theobject deformations, if possible. Based upon this discovery, the presentdisclosure systems and methods would have to perform much better incorrecting the errors in the deformations.

Still other test approaches included applications that included stepsnecessary to recover a signal observed through multiple snapshots, suchthat each underwent an unknown or partially known deformation or ascrambling, i.e., a permutation of the signal. In this case theobjective was to recover the permutation, in addition to recovering thesignal. This proved to be a difficult problem, as the number of possiblepermutations increased exponentially with the size of the signal. In anumber of test applications, certain permutations were more likely thanothers. It would be desirable to be able to exploit this information toreduce the difficulty of the problem. However, in the current state ofthe art it is not known how this information can be used effectively.

Still some other test approaches were developed to analyze imaging of adeformable moving object using Inverse Synthetic Aperture Radar, ISAR.However, what was later discovered is that these systems cannot considerdeformations of the object, for example, hands moving while a person iswalking or heart beating of a person. What was also learned is thatthese test approaches also do not consider errors in the model of theobject motion. Which these test approaches when addressing these errorsnecessitate techniques that were very computationally cost expensive orrobust to errors, such as incoherent imaging in the case of radar, whichcompromise imaging quality.

Other test approaches included recovering a signal observed throughunknown permutations. However, during the testing process we realized noknown methods disclose recovery of a permuted signal measured through ameasurement system. In these particular test approaches it becameevident that known methods only consider direct observation of thepermuted signal. What was realized is that adding a measurement systemis not obvious because a measurement system combines elements of thepermuted signal. Some methods used in these particular test approachessimply will not work if the elements of the signal are combined intomeasurements by the measurement system of some embodiments of thepresent disclosure. Furthermore, these particular methods used in testapproaches cannot exploit the knowledge on the permutation matrix, i.e.that permutations that move image pixels closer are more likely thanpermutations that move image pixels farther.

Some methods in the test approaches resulted in providing somecorrection of the deformation using the measurements. However, in thosetest cases, the computation was simplistic and often failed. Gained fromthese test cases is that some embodiments of the present disclosureexploit formulations that provides the use of optimal transport theoryand algorithms to correctly estimate both the deformations and theircorrections.

Some test approaches combine information from different modalities. Atleast some problems with the test approaches is that the sensor(s) inthe modality (or modalities) used for tracking, made errors and had alower resolution than that which the imaging sensor required. These testapproaches/applications assumed the errors away, which resulted toinferior performance. Gained from these test approaches, is that someembodiments of the present disclosure are configured to provide acorrection of the tracking that is in a higher resolution than requiredby the imaging sensor(s).

Furthermore, some important realizations gained from experimentation,was that both problems, namely imaging of deformable objects underdeformations and that recovering a signal observed though unknownpermutation, can be expressed using the same underlying formulation.Having completed extensive experimentation, this new knowledge is notobvious because these are two very different problems with verydifferent applications. The former, imaging of deformable objects underdeformations has applications in medical imaging and security screening,among others, while the latter, recovering a signal observed thoughunknown permutation has applications in unlabeled and partially labeledsampling and simultaneous localization and mapping (SLAM), among others.

Thus, this formulation incorporated in some embodiments of the presentdisclosure includes

-   -   (1) an unknown signal x being measured by taking one or more        snapshots;    -   (2) a linear transformation of the signal being measured in any        snapshot (F_(i));    -   (3) an unknown permutation that affects the signal (P_(i));    -   (4) a measurement system (A_(i)), that may or may not be the        identity system which directly measures the signal; and    -   (5) a set of measurements y_(i) of the unknown transformed        permuted signal.

Another important realization of the present disclosure is that thisformulation can be further relaxed, to allow for softer solutions. Whichthis can allow to compute the gradient of the cost function, whichprovides its optimization using gradient-based algorithms. The cost isdiscrete without the relaxation, and therefore has no gradient. Theoptimization is combinatorial in that case, which has prohibitivecomputational complexity for any problem of reasonably practical size.

Another realization is that this particular choice of relaxationprovides for the use of efficient methods based on optimal transport,which are able to provide better solutions and are more likely toconverge to a good optimum. The problem is non-convex and, therefore,naïve relaxations end up exhibiting too many local minima, and notproviding good solutions to the problem. Another important realizationis that the permutation matrices P_(i) do not need to be estimatedexplicitly and only an estimate of the signal x is required. Thisfurther provides the use of optimal transport methods, which provide a“transport plan” which implicitly estimates the permutation.

Another important realization is that when the problem is relaxed asdescribed above, it becomes a bilinear problem. Thus, the problem can beefficiently solved using alternating minimization, where the algorithmalternates between estimating the original signal x and estimating thepermuted transformed signals x_(i) that the measurement systems measuresin each snapshot.

According to an embodiment of the present disclosure, an imaging systemincluding a tracking system to track a deforming object within a sceneover multiple time steps for a period of time to produce an initialestimate of a deformation of the object moving for each time step. Ameasurement sensor captures measurements of the object deforming in thescene over the multiple time steps for the time period as measurementdata, by capturing snapshots of the object moving over the multiple timesteps. A processor that calculates, for the measurement data,deformation information of the deforming object. Each acquired snapshotof the object includes measurements of the object in a deformation forthat time step in the measurement data, to produce a set of measurementsof the object with deformed shapes over the multiple time steps. Foreach time step of the multiple time steps, the processor sequentiallycalculates deformation information of object, by computing a correctionto the estimates of the deformation of the object. Such that thecorrection includes matching measurements of the corrected deformationof the object for each time step to measurements in the acquiredsnapshot of the object for that time step. Wherein for each time step, acorrected deformation is selected over other corrected deformations forthat time step, according to a distance between the correcteddeformation and the initial estimate of the deformation, to obtain afinal estimate of the deformation of the deformable object moving in thescene and a final image of the object moving within the scene.

According to another embodiment of the present disclosure, an imageprocessing method including tracking a deforming object while movingwithin a scene over multiple time steps for a period of time via atracking system to produce an initial estimate of a deformation of theobject for each time step. Acquiring measurement data by continuouslycapturing snapshots of the object deforming in the scene over themultiple time steps for the period of time. Computing deformationinformation of the deforming object, by producing a set of measurementsof the object with deformed shapes over the multiple time steps, fromeach acquired snapshot of the object that includes measurements of theobject in a deformation for that time step in the measurement data.Calculating deformation information of object, by computing a correctionto the estimates of the deformation of the object for each time step forthe multiple time steps. Wherein the computing of the correctionincludes matching measurements of the corrected deformation of theobject for each time step to measurements in the acquired snapshot ofthe object for that time step. Wherein for each time step, selecting acorrected deformation over other corrected deformations for that timestep, according to a distance between the corrected deformation and theinitial estimate of the deformation, to obtain a final estimate of thedeformation of the deformable object moving in the scene and a finalimage of the object moving within the scene, which are stored.

According to another embodiment of the present disclosure, a productionapparatus including a tracking system to track a deforming object withina scene over multiple time steps for a period of time to produce aninitial estimate of a deformation of the object for each time step. Ameasurement sensor including an electromagnetic sensor capturesmeasurement of the object deforming in the scene over the multiple timesteps for the time period as measurement data, by capturing snapshots ofthe object moving over the multiple time steps. A processor calculates,for the measurement data, deformation information of the deformingobject. Each acquired snapshot of the object includes measurements ofthe object in a deformation for that time step, to produce a set ofmeasurements of the object with deformed shapes over the multiple timesteps in the measurement data. For each time step of the multiple timesteps, the processor sequentially calculates deformation information ofobject, by computing a correction to the estimates of the deformation ofthe object for each time step for the multiple time steps. Wherein thecorrection includes matching measurements of the corrected deformationof the object for each time step to measurements in the acquiredsnapshot of the object for that time step. Wherein for each time step, acorrected deformation is selected over other corrected deformations forthat time step, according to a distance between the correcteddeformation and the initial estimate of the deformation, to obtain afinal estimate of the deformation of the deformable object moving in thescene and a final image of the object moving within the scene, which arestored.

According to another embodiment of the present disclosure, a radarsystem. The system including a tracking system tracking a deformingobject while moving within the scene over multiple time steps for aperiod of time to produce an initial estimate of a deformation of theobject moving for each time step, such that at each time step includes adifferent deformation. A sensor captures measurements of the objectdeforming in the scene over the multiple time steps for the time periodas measurement data, by capturing snapshots of the object moving overthe multiple time steps. A processor that calculates, for themeasurement data, deformation information of the deforming object. Eachacquired snapshot of the object includes measurements of the object in adeformation for that time step, to produce a set of measurements of theobject with deformed shapes over the multiple time steps in themeasurement data. For each time step of the multiple time steps, theprocessor sequentially calculates deformation information of object, bycomputing a correction to the initial estimates of the deformation ofthe object for each time step for the multiple time steps. Such that thecorrection includes matching measurements of the corrected deformationof the object for each time step to measurements in the acquiredsnapshot of the object for that time step. Wherein for each time step, acorrected deformation is selected over other corrected deformations forthat time step, according to a distance between the correcteddeformation and the initial estimate of the deformation, to obtain afinal estimate of the deformation of the deformable object moving in thescene and a final image of the object moving within the scene. An outputinterface outputs the final estimate of the deformation of thedeformable object, the final image of the object moving within thescene, or both, to one or more components of an output interface of theradar system, or to another system or a communication network associatedwith the radar system.

According to another embodiment of the present disclosure, a radarimaging method to reconstruct a radar reflectivity image of a scene.Tracking a deforming object while moving within the scene over multipletime steps for a period of time with a tracking system, to produce aninitial estimate of a deformation of the object for each time step ofthe multiple time steps. At least one electromagnetic sensor capturesmeasurements of the object deforming in the scene over the multiple timesteps for the time period as measurement data, by capturing snapshots ofthe object moving over the multiple time steps. Each acquired snapshotof the object includes measurements of the object in a deformation forthat time step, to produce a set of measurements of the object withdeformed shapes over the multiple time steps in the measurement data.The method including using a processor for calculating deformationinformation of object, by computing a correction to the estimates of thedeformation of the object for each time step for the multiple timesteps. Such that the computing of the correction includes matchingmeasurements of the corrected deformation of the object for each timestep to measurements in the acquired snapshot of the object for thattime step. Wherein for each time step, selecting a corrected deformationover other corrected deformations for that time step, according to adistance between the corrected deformation and the initial estimate ofthe deformation, to obtain a final estimate of the deformation of thedeformable object moving in the scene and a final radar image of theobject deforming within the scene. Outputting one or a combination ofthe final estimate of the deformation of the deformable object or thefinal radar image of the object, to one or more components of the radarsystem or another system associated with the radar system.

According to another embodiment of the present disclosure, anon-transitory computer readable storage medium embodied thereon aprogram executable by a processor for performing a radar imaging method.The radar imaging method is to reconstruct a radar reflectivity image ofa scene including an object deforming within the scene. Tracking thedeforming object that deforms over multiple time steps for a period oftime using a tracking system having an optical sensor to produce aninitial estimate of a deformation of the object for each time step ofthe multiple time steps. Acquiring measurement data by continuouslycapturing snapshots of the object deforming in the scene over themultiple time steps for the period of time. Such that, at each time stepincludes a different deformation. The method including computingdeformation information of the deforming object, by producing a set ofmeasurements of the object with deformed shapes over the multiple timesteps, from each acquired snapshot of the object that includesmeasurements of the object in a deformation for that time step in themeasurement data. Calculating deformation information of object, bycomputing a correction to the estimates of the deformation of the objectfor each time step for the multiple time steps. Wherein the computing ofthe correction includes matching measurements of the correcteddeformation of the object for each time step to measurements in theacquired snapshot of the object for that time step. Wherein for eachtime step, selecting a corrected deformation over other correcteddeformations for that time step, according to a distance between thecorrected deformation and the initial estimate of the deformation, toobtain a final estimate of the deformation of the deformable objectmoving in the scene and a final radar image of the object deformingwithin the scene, which are stored. Outputting the final estimate of thedeformation of the deformable object within the scene, the final radarimage of the object within the scene, or both, to one or more componentsof the radar system or a communication network associated with the radarsystem.

BRIEF DESCRIPTION OF THE DRAWINGS

The presently disclosed embodiments will be further explained withreference to the attached drawings. The drawings shown are notnecessarily to scale, with emphasis instead generally being placed uponillustrating the principles of the presently disclosed embodiments.

FIG. 1A is a schematic illustrating of a radar imaging system todetermine a radar reflectivity image of an object moving within a scene,according to an embodiment of the present disclosure;

FIG. 1B is a flow diagram illustrating some method steps forimplementing a method, according to some embodiments of the presentdisclosure;

FIG. 2A is a schematic illustrating deformation of the object beingimaged, according to some embodiments of the present disclosure;

FIG. 2B is a schematic illustrating some components that can be utilizedwith a radar imaging system, according to some embodiments of thepresent disclosure;

FIG. 2C is a schematic illustrating an MRI machine utilized with a radarimaging system to scan a person, according to some embodiments of thepresent disclosure;

FIG. 3 is a schematic illustrating of dual-grid representation of anobject, according to some embodiments of the present disclosure;

FIG. 4 shows a schematic capturing the motion of the object using thedual-grid representation, according to some embodiments of the presentdisclosure;

FIG. 5 shows a schematic capturing the transformation of the objectcaused by its motion using the dual-grid representation, according tosome embodiments of the present disclosure;

FIG. 6 shows a schematic of an electromagnetic sensor, such as a radar,acquiring the radar reflectivity image, according to some embodiments ofthe present disclosure;

FIG. 7 shows a schematic of reconstruction of a radar reflectivityimage, according to some embodiments of the present disclosure;

FIG. 8 shows an example of the motion and deformation of the object infront of the optical and radar sensors at each snapshot, according tosome embodiments of the present disclosure;

FIG. 9 shows a schematic of the tracking performed by the optical sensorusing the example of FIG. 8, according to some embodiments of thepresent disclosure;

FIG. 10A shows a flowchart of the optimization procedure used to recoverthe deformations and the measured signal, according to some embodimentsof the present disclosure;

FIG. 10B, FIG. 10C and FIG. 10D show pseudocode implementing aspects ofthe flowchart in FIG. 10A, according to some embodiments of the presentdisclosure;

FIG. 11A to FIG. 11E show experimentation performed on exampleembodiments, FIG. 11A shows a signal x in a prototypical position, FIG.11B shows a F_(i)x 1^(st) estimated deformation of the snapshot, FIG.11C shows a F_(i)x 2^(nd) estimated deformation of the snapshot, FIG.11D and FIG. E show actual deformations x_(i)=P_(i)F_(i)x of x observedby the acquisition system, according to some embodiments of the presentdisclosure;

FIG. 12A and FIG. 12B show performance analysis of some embodiments forvarious experimental conditions and comparisons with conventionalaspects for the experimental example in FIG. 11A to FIG. 11E, accordingto some embodiments of the present disclosure;

FIG. 13 shows a hardware diagram of different components of the radarimaging system, according to some embodiments of the present disclosure;and

FIG. 14 is a schematic illustrating a computing apparatus that can beused to implement some techniques of the methods and systems, accordingto some embodiments of the present disclosure.

DETAILED DESCRIPTION

While the above-identified drawings set forth presently disclosedembodiments, other embodiments are also contemplated, as noted in thediscussion. This disclosure presents illustrative embodiments by way ofrepresentation and not limitation. Numerous other modifications andembodiments can be devised by those skilled in the art which fall withinthe scope and spirit of the principles of the presently disclosedembodiments.

FIG. 1A is a schematic illustrating of an imaging system 100A todetermine a reflectivity image of an object moving within a scene 105,according to an embodiment of the present disclosure. The imaging system100A can include at least one tracking sensor 102, such as an optical oran ultrasonic sensor, configured to acquire optical reflectivity imagesof the scene 105 and at least one measurement sensor 104, configured toacquire measurements of the scene 105. Examples of the tracking sensor102 include one or combination of an optical camera, a depth camera, aninfrared camera, and an ultrasonic sensor. Examples of the measurementsensor 104 include one or combination of a millimeter wave (mmWave)radar, a terahertz (Thz) imaging sensor, a backscatter X-ray, magneticresonance imaging, and a tomographic X-ray sensor.

The tracking sensor 102 can be configured to track the object in thescene 105 over multiple time steps in a period of time to produce, foreach of the multiple time steps, a shape of the object at a current timestep. In various embodiments, the tracking sensor 102 can determine theshape of the object as an inexact deformation 115 of a nominal shape ofthe object, wherein the deformation is inexact because it may containtracking errors, or might not exhibit the tracking resolution necessaryto reconstruct the object in the modality of the measurement sensor,using the measurements of the measurement sensor. For example, thenominal shape of the object may be a shape of the object arranged in aprototypical pose typically known in advance. In other embodiments thetracking sensor 102 can determine the shape of the object in one-timestep as an inexact deformation 115 of a shape of the object in adifferent time step, wherein the deformation is inexact because it maycontain tracking errors, or might not exhibit the tracking resolutionnecessary to reconstruct the object in the modality of the measurementsensor, using the measurements of the measurement sensor.

Still referring to FIG. 1A, the measurement sensor 104 can be configuredto acquire snapshots of the scene 105 over the multiple time stepswithin the time period to produce a set of measurements 117 of theobject with deformed shapes defined by corresponding deformations of theshape of the object determined by the tracking sensor 102. Notably, dueto the movement of the object in the scene 105, at least two differentmeasurement snapshots can include the object with different deformedshapes.

The imaging system 100A can include at least one processor 107. Theprocessor 107 can be configured to determine 111, for each snapshot ineach time step of the multiple time steps, a correction of thedeformation 115 determined for the corresponding time step, whichincorporates the measurements of the scene at the time step, to producean accurate deformation using embodiments of the present disclosure. Theprocessor may further be configured to determine the image of the objectin the modality of the measurement sensor, under a particulardeformation, incorporating the correction of the deformation in one ormore time-steps and the measurement snapshots in one or more time-steps.

Still referring to FIG. 1A, because the object is moving in the scene105, different measurement snapshots can be obtained from differenttransformations for each time step of the multiple time steps. In someembodiments, the tracking and the measurement snapshots can besynchronized, e.g., taken concurrently at corresponding time stepsand/or with a predetermined time shift, such that the image of theobject is determined using the deformation produced by the correspondingtracking sensor, synchronized or acquired at the same time step.

In some embodiments, the tracking and the measurement sensor may be thesame sensor, wherein the processor 107 is further configured todetermine the inexact deformation before computing a correction. Inother embodiments, the tracking sensor may or may not be the same sensoras the measurement sensor, and the processor directly computes anaccurate deformation incorporating tracking snapshots in one or moretime steps and the measurement snapshots in one or more time steps.

Still referring to FIG. 1A, in some embodiments, the processor mayfurther incorporate other available information in determining theinexact and the accurate deformation for each time step. Thisinformation may include, but it not limited to:

-   -   i) the position of the sensor system at the time of each        snapshot;    -   ii) the orientation and field of view of each of the sensors at        each time step;    -   iii) prior measurements of the scene or knowledge of the scene        geometry from existing sources, such as maps and wireframe        representation and images,    -   iv) dynamic information of the object deformation, such as heart        rate and beating models, lung breathing rate and deformation        models, etc.;    -   v) odometry of the platform on which the sensor is mounted if        the sensor is mobile, including velocity, acceleration, pitch,        yaw, and kinematic motion models;    -   vi) pre-determined reflectivity patterns or marking, such as QR        codes, corner reflectors, or motion capture markers;    -   vii) pre-existing sensor landmarks in the scene and their exact        geometry,        along with any other information that may assist the processor        in determining a deformation of the object

Still referring to FIG. 1A, some embodiments can be based on recognitionthat an imaging system can jointly use the measurements of a sceneacquired over multiple time steps. Such a system of measurements can beused to improve the resolution of the measured image beyond a resolutiongoverned by the size of the imaging system, known in the art as aperturesize. When the object is moving or deforming over time, at differenttime steps, the object can be located at a different position and canhave a different shape caused by the non-rigid motion. Such adislocation and deformation of the object make the system ofmeasurements ambiguous, i.e., ill-posed, and difficult or impractical tosolve. However, the embodiments can, instead, exploit the diversityintroduced by the motion by determining and using the transformationsbetween each measurement snapshot to construct a larger syntheticaperture, which allows for higher effective resolution. This in known inthe art as inverse synthetic aperture imaging (ISAR). However, themethods in the art are not able to incorporate deformable objects in thetracking.

Some embodiments of the present disclosure provide ISAR for deformableobjects. Thus, the embodiments can jointly use measurements of a sceneacquired over multiple time steps to produce the image of the object inone or more specific poses or deformation. For example, the image of ahuman may be reproduced as the human is walking through the system or ina pose wherein all parts of the human body are visible and not occluded.As another example, an image of a beating heart or lungs may bereproduced at a predetermined phase of the beating or the breathingpattern.

Still referring to FIG. 1A, some embodiments are based on recognitionthat at least one reason for using separate sensors of to measure and totrack a moving object can be due to a resolution of the measurementsensor which is governed by a physical size of the antennas of thesensors. Specifically, for practical reasons, the size of the antennasof an imaging system can allow the estimation of only a coarse image ofthe object at each time step. Such a coarse image can be suitable totrack an object subject to rigid and finite transformation; however,this type of component configuration can fail to recognize arbitrarilynon-rigid transformation typical for the motion of a human.

Some embodiments are based on recognition that other sensors, such asoptical monochrome or color or infrared video cameras, or depth cameras,or ultrasonic sensors, or a combination thereof, are cheaper than themeasurement sensor with comparable resolution and also more suitable fortracking. Hence, a tracking sensor can be used for tracking the motionof the target, even if the target is deformable and the motion is notrigid. On the other hand, tracking sensors, using a different modalitythan the measurement sensors, might not be able to provide theinformation or the resolution necessary for the function of the sensingsystem. For example, optical sensors are not able to see coveredobjects, and, thus are not able to detect dangerous weapons orcontraband in a security screening application, even though they can beused to track a human moving through the system. Similarly, ultrasonicsensors are very inexpensive and are able to detect and track a beatingheart or a lung breathing pattern. However, they are not sufficientlyprecise to image the beating heart or the lung with the same resolutionand fidelity as an MRI or CAT system.

Some embodiments are based on realization that for a number ofapplications, it is sufficient to determine a radar reflectivity imageof an object at some prototypical pose, not necessarily at a currentpose that object has at a current instance of time. For example, forsome security applications, the prototypical pose of a person isstanding, with the hands extended upwards or sideways. The objectarranged in the prototypical pose has a nominal shape that can change,i.e., deform, as the object moves.

Still referring to FIG. 1A, some embodiments are based on anotherrealization that the optical sensor can track the object in the sceneusing the relative deformation to the previous pose, rather than thedeformation from a common, prototypical pose. For example, instead ofdetermining a pose and/or an absolute current shape of the object in thecurrent snapshot of the scene, the optical sensor can determine therelative shape of the object as a deformation a shape of the object inanother snapshot. The embodiments are based on realization that thedeformation of the nominal shape of the object determined by thetracking sensor can be used to reconstruct the image of the object insome pose.

FIG. 1B is a flow diagram illustrating some method steps forimplementing a method, according to some embodiments of the presentdisclosure. The method starts 118 and obtains multiple snapshots of asignal of interest 120, which include a subset or a combination oftracking data and measurements of the signal of interest that can beused to reconstruct an image of the signal of interest. The signal ofinterest may include a specific object fully or partially in the fieldof view of the sensors, a scene, or a combination thereof, that hasundergone a deformation in each of the snapshots.

In some embodiments, if possible by the available tracking data andmeasurements, an estimate of the approximate deformation of the signalof interest in each of the snapshot is computed 122, using methods knownin the art. A cost function 124, relating, among other possiblyavailable information, the true deformation of the signal of interest,the approximate estimate of the deformation, the signal of interest, themeasurements of the signal of interest and the tracking data, is reducediteratively 127, until convergence 126, as described below.

If required, in some embodiments, the computed deformations are used toreconstruct the signal of interest 128. The signal of interest or thecomputed deformations, or both are output 132 by the method, as requiredby the application and further processing steps.

Still referring to FIG. 1B, Contemplated is that some steps of a radarmethod to estimate a deformation of a deformable object moving in ascene that can include steps of, for example, tracking a deformingobject within the scene over multiple time steps for a period of timevia a tracking system with a tracking sensor to estimate a deformationof the object for each time step. A step of using an electromagneticsensor(s) that captures measurements of the object deforming in thescene over the multiple time steps for the time period as measurementdata, by capturing snapshots of the object moving over the multiple timesteps. Another step using a processor that calculates, for themeasurement data, deformation information of the deforming object. Whichcan include the electromagnetic sensor that captures snapshots of theobject deforming over the multiple time steps. Each acquired snapshot ofthe object in the measurement data includes measurements of the objectin a deformation for that time step, to produce a set of measurements ofthe object with deformed shapes over the multiple time steps. Wherein,for each time step for the multiple time steps, the processorsequentially calculates deformation information of object, by computinga correction to the estimates of the deformation of the object for eachtime step for the multiple time steps.

A step where the correction can include matching measurements of thecorrected deformation of the object for each time step to measurementsin the acquired snapshot of the object for that time step. Wherein, foreach time step, select a corrected deformation over other correcteddeformations for that time step, according to a distance between thecorrected deformation and the initial estimate of the deformation, toobtain a final estimate of the deformation of the deformable objectmoving in the scene and a final image of the object moving within thescene.

Depending upon a user or operator's specific goals, a step can includeoutput the final estimate of the deformation of the deformable object toone or more components of at least one output of the radar system or toanother system associated with the radar system.

FIG. 2A is a schematic illustrating relative tracking utilizing at leastone measurement device such as an optical sensor, according to someembodiments of the present disclosure. For example, some embodiments cantrack a moving object, e.g., a person, by determining at each timeinstant or time step, a deformation 220 of a nominal shape 210 of theobject resulting in a deformed shape 230 of the object.

Some embodiments are based on a realization that the deformation 220indicative of a transformation of an object in a tracking modality isalso indicative of the transformation of the object in the measurementmodality, even if the two modalities are different. Therefore, anapproximate deformation can be computed from the tracking sensor output.

FIG. 2B is a schematic illustrating some components that can be utilizedwith a radar imaging system, according to some embodiments of thepresent disclosure. For example, an object 240 can be person walking ormoving relative to a sensor configuration, in which, the person 240 canbe imaged by measurement sensors. Some types of measurement sensors caninclude radar, mmWave, backscatter X-ray or THz sensors 260A-260C,261A-261C, 262A-262C, which for each time step can take snapshots inorder to generate an image of the person in some pose, which may be athree-dimensional (3D) image, wherein the pose may be a pose of theperson in one of the time steps or a pose the person never used, such asa prototypical pose with the arms out and stretched so that all sides ofthe body are visible.

Some types of tracking sensors may include optical and depth sensors265A-265C that may additionally detect a three-dimensional 3D model ofthe object 240, i.e. person, in order to track the deformations of theperson as it moves through the imaging system. For example, tracking thedeformation of the object may include determining the position andorientation of each part of the person's body, such as the arms andlegs, relative to the imaging system. It may also include using awireframe model of the body and, for an acquired snapshot, determiningthe location within the sensing system of every point of the wireframemodel and/or a determination whether that point is occluded to thecamera at the time step of that snapshot. Tracking the deformation ofthe body may also include mapping pixels or voxels from one snapshot toanother, such that pixels from one snapshot mapped to pixels fromanother snapshot correspond to the same part of the body as it has movedbetween the two snapshots.

Still referring to FIG. 2B, operationally, the measurement sensors ofthe sensor configuration generate data to be sent to a hardwareprocessor, such as the hardware processor 107 of FIG. 1A to beprocessed. The person 240 moves relative to the sensor configuration,where, for example, a transmitter of the measurement sensors 260A-260C,261A-261C, 262A-262C may emit continuously or in pulses, a transmissionsignal. The waves of the transmission signal are reflected on the personor object 240, and are detected by a receiving sensor. Some measurementsensors may be passive, i.e., only have receiving sensors. The signal isdigitized and routed to the hardware processor, i.e. for example theprocessor 107 of FIG. 1A. In respect to the sensor configuration, otherembodiments are contemplated including more sensors, differentcombination of types of sensors and even combinations of types of radarsystems. Depending upon a user specific goals and sensor configurationrequirements other components can include higher performance componentsprovided in FIG. 13 and FIG. 14. The different types of sensors of thesensor configuration can be synchronized with sufficiently high accuracyso that they work coherently together. Each receiving unit can beequipped with a receiving channel and analog-to-digital conversionsystems.

Rotating or otherwise mobile structures 256, 258 may be configured withdifferent types of sensors in order to address specific user goals andapplication requirements. As an example, these rotating structures canrotate in a clockwise direction D1, D2, D3, D4, or a counter clockwisedirection (not shown), depending upon the user specific requirements.Also, the rotating structures 256, 258 may be placed on rails to eitherincrease or decrease the rotating structure height (not shown), or eventravel on rails (not shown) along an horizontal axis H and/or y axis Y.Some aspects as to why the sensor configuration can include multiplemovement characteristics can be associate with user's specificapplication requirements. For example, a user may utilize the sensorconfiguration for security related applications including airport,building, etc. to identify potential weapons, and the like. Wherein a360° imaging of the object is less expensive with the measuring sensorspositioned on the rotating structures 256, 258, as it requires fewersensors. Contemplated is that other types of sensors, i.e. audio,temperature, humidity, etc., along with lighting, can be mounted on therotating structures 256, 258 and the other structures A, B, C. Somebenefits of using the rotating structures 256, 258 can include a largertarget area that can be covered by the measuring sensors and a largereffective aperture, which provide a higher resolution image.

FIG. 2C is a schematic illustrating a MRI or a CAT machine 270 utilizedas an imaging system to scan a patient/person 272 on a table 274,according to some embodiments of the present disclosure. MRI or CATmachines are used to image the internal structure of the body to detectmedical conditions and assist physicians' diagnoses. The machine takesmultiple snapshots of the body being imaged under different angles, thuscreating a synthetic aperture imaging the area of interest inside thebody.

To ensure that synthetic aperture imaging reconstructs the correctimage, without blurring or motion artifacts, the patient should be keptas still as possible during imaging. This is a problem, especially whenimaging moving and deforming organs, such as the heart or the lung. Insuch applications, embodiments of the present disclosure may use one ormore tracking sensors, which may include but not limited to anultrasonic sensor, a heart rate monitor, or a breathing rate sensor,among others.

Still referring to FIG. 2C, as the organ being imaged moves and deformsduring imaging, the tracking sensor estimates an inexact deformation ofthe organ for each of the snapshot taken by the measurement system.Using some embodiments of present disclosure it is, thus possible todetermine an image of the organ under any deformation, even though eachsnapshot is taken in a different deformation. Using some embodiments ofthe present disclosure, the processor 107 in FIG. 1A determines acorrection of the inexact deformation and an image of the organ in theimaging modality, considering the measurements for each snapshot, asdescribed in the present disclosure.

FIG. 3 is a schematic illustrating of dual-grid representation of anobject, according to some embodiments of the present disclosure. Forexample, a deformable object 310, a human in this example, is in aprototypical pose. To construct a radar reflectivity image in theprototypical form, a grid 320 can be defined on the prototypical pose ofthe object. In other words, the first grid 320 of the dual-gridrepresentation is a prototypical grid that discretizes the objectitself. For example, the grid in the image has grid positions 330indexed as 1, 2, . . . , N. The radar reflectivity of each point in thegrid 340 is denoted as x₁, x₂, . . . , x_(N). There are several ways toindex the prototypical grid, but in general they can always be mapped toa sequential grid, as shown in the figure. Any grid indexing approachcan be used in embodiments of the present disclosure.

The second grid of the dual-grid representation is a radar grid thatdiscretizes the scene itself. For example, in one embodiment the secondgrid is a rectangular (Cartesian) grid 550. However, other grids, suchas a radial one may also be used by different embodiments. As with theprototypical grid, there are several ways to index the radar grid usedby different embodiments. For example, in the embodiment shown in FIG.3, the radar grid is indexed using Cartesian coordinates 360. Themeasurement sensor, e.g., radar 370, and/or individual radartransceivers 380 have positions inside the radar grid, at knownlocations.

Still referring to FIG. 3, in some embodiments, both grids in thedual-grid representation are the same, e.g., they are both cartesiangrids. This representation is particularly useful when the deformationis determined as a deformation of the pose of the object between twosnapshots, since the representation of the first grid corresponds to thegrid of the first snapshot and the representation of the second gridcorresponds to the grid of the second snapshot.

FIG. 4 shows a schematic capturing the deformation of the object usingthe dual-grid representation, according to some embodiments of thepresent disclosure. FIG. 4 illustrates the object in the pose at thefirst grid 400, as well as the object's pose in the second grid in frontof the measurement sensor in a single snapshot 450. The object's pose infront of the measurement sensor can be described by a deformation 440 ofthe object in the first grid to represent the deformation of the objectin the second grid. The object in the second grid is observed by themeasurement sensor, e.g., a radar, 470 and its individual transceivers480 according to a measurement operator related to the hardware of themeasurement sensor. As described above, the deformation of the radargrid may be inferred by the tracking sensor, which might be the same asthe measurement sensor, or a tracking model or other information.

FIG. 5 shows a schematic capturing the transformation of the objectcaused by its motion using the dual-grid representation, according tosome embodiments of the present disclosure. This embodiment determineseach transformation as a subsampled permutation that permutes locationsof some points of the image of the object in the first grid and removesdifferent points of the image of the object in the first grid, independence of the deformation of the nominal shape of the object in thefirst grid.

Referring to FIG. 4 and FIG. 5, specifically, the deformation is asubsampled coordinate permutation 545 of FIG. 5, i.e., a transformationthat maps the indices in the coordinate system of the first grid to theindices of the second grid. Thus, the image of the object measured bythe measurement sensor is a simple permutation with erasures that maps560 of FIG. 5 the image of the object in the first grid 410 of FIG. 4 tothe image of the object in the second grid, consistent with the object'spose.

Still referring to FIG. 5, more generally, in some embodiments, theimage of the deformed object in the second grid is a lineartransformation of the image of the object in the prototypical pose, thatcan be described as

z=Fx,  (1)

where x is the image of the object in the pose in the first grid and zis the image of the deformed object in the radar grid.

FIG. 6 shows a schematic of a measurement sensor, using a radar as anexample, acquiring the radar reflectivity image, according to someembodiments of the present disclosure. The sensor 670 includestransceivers 680 (or separate transmitters and receivers) which are usedto sense the scene. In particular, one or more of the transceiverstransmit a pulse 620 or otherwise excite the scene. The pulses areabsorbed by or reflect back from the object 650, according to thereflectivity of the object. The reflected pulses 630 are acquired by oneor more transceivers 680. The pulse transmission and acquisition, iscontrolled by a radar control, pulsing and acquisition system 690, whichmay control the pulse shape and timing, as well as which transceiverstransmit and which receive. In some modalities, such as MRI, instead ofreflecting, the excited object might generate its own signal, such as aresonance according to the response properties of the object. Ingeneral, these response properties, which may include a reflectivity,comprise the image of the object.

The system is configured for acquiring the signals that the receiversreceive in response to the received pulses from the scene, for example,using a data acquisition system. A data acquisition system may includeone or more amplifiers, one or more modulators, and one or moreanalog-to-digital converters, among others. The system outputs data y695 which represent recordings of the pulse reflections. Theserecordings are samples of the reflections or a function of them, such asdemodulation, filtering, de-chirping, or other pre-processing functionsknown in the art. This data comprises the measurements of the scene ineach snapshot.

Still referring to FIG. 6, the acquired data y are linear measurementsof z, the radar reflectivity image of the deformed object 650 in theradar scene, through the radar acquisition function, also known in theart as the forward operator denoted here using A. Thus, the acquireddata for a single snapshot are equal to

y=Az=AFx.  (2)

If the radar system has a sufficient number of sensors and a bigaperture, then the data y may be sufficient to recover z, the radarreflectivity image of the object in the deformed pose. However,recovering the image in high resolution would require a large andexpensive radar array. Furthermore, in particular deformations, parts ofthe object might not be visible to the array, which can make their radarreflectivity not recoverable, irrespective of the radar array size.

Still referring to FIG. 6, for that reason, an imaging system of someembodiments acquire measurements of several snapshots of the image,under different deformations

y _(i) =A _(i) z _(i) =A _(i) F _(i) x,  (3)

where i=1, . . . , T is the index of the snapshot, and T is the totalnumber of snapshots. In various embodiments, the only change betweensnapshots is the deformation of the object, and, therefore, thedeformation F _(i) of the radar reflectivity image. In some embodiments,the measurement operator can be different when, for example, differenttransducers are used to acquire each snapshot, or the sensor moves orrotates on a moving platform. In other embodiments the forward operatoris always the same in each snapshot, in which case A_(i)=A for all i.

If all the deformations are perfectly known, the image of the object canbe reconstructed by combining the measurements of images of the objectwith deformed shapes transformed with the corresponding transformations.For example, using multiple snapshots, the reconstruction problembecomes one of recovering x from

$\begin{matrix}{{\begin{bmatrix}y_{1} \\\vdots \\y_{T}\end{bmatrix} = {\begin{bmatrix}{A_{1}{\overset{¯}{F}}_{1}} \\\vdots \\{A_{T}{\overset{¯}{F}}_{T}}\end{bmatrix}x}},} & (4)\end{matrix}$

which, assuming the F _(i) are known, can be performed using, e.g.,least squares inversion. Solutions of (4) in the art may imposeadditional regularization constraints on reconstructing image x, such assparsity or smoothness, by expressing x in some lower-dimensional basisof a subspace or a large dictionary, i.e., x=Bh, where B is a lowdimensional basis or a dictionary, and h is a set of coefficients eitherlower-dimensional than x or sparse. Alternatively, or additionally,other solutions in the art may impose low total variation structure onx, i.e., sparsity in its gradient. All these regularization constraintscan be imposed by regularizing using different techniques.

Still referring to FIG. 6, some embodiments determine the deformation F_(i) of the image from the deformation of an image acquired in adifferent modality, such as an optical image. In other words, by using adifferent modality it is possible to infer the physical deformation ofthe object. Since the reflectivity of each point in the object's imagein the measurement modality does not change with the position of thepoint, i.e., with the deformation of the object, the deformationinferred by the sensor of the different modality can be used to inferthe deformation of the object's image in the measurement modality.

Optical sensors, such as monochrome, color, or infrared cameras recordsnapshots of the reflectivity of objects as they move through a scene.Using two or more of these cameras, placed at some distance apart, it ispossible to determine the distance of each point of the object from eachcamera, known in the art as depth of the point. Similarly, depth camerasuse the time-of-flight of optical pulses or structured light patterns todetermine depth. By acquiring the optical reflectivity and/or the depthof the object as it moves, there are methods in the art to track thepoints of the object, i.e., to determine, in each snapshot, thedeformation of the objects from the deformation of the optical or thedepth image. Determining this deformation is possible in the art, eventhough the optical reflection of the object changes with deformation dueto lighting, occlusion, shadowing and other effects.

Still referring to FIG. 6, thus, an optical sensor, such as a camera ora depth sensor, can be used to infer the deformation F _(i) in eachsnapshot. The optical sensor acquires a snapshot of the object at thesame time instance as the radar sensor acquires a snapshot of the radarreflectivity image, as described in (3). This radar reflectivity imagecan then be used to track the deformation of the object in order toreconstruct its radar reflectivity image. In some embodiments, theoptical sensor might acquire snapshots at different time instances thanthe radar sensor. The deformation of the object at the time instancethat the radar acquires the snapshot is can then be inferred usingmultiple optical snapshots, using techniques known in the art, such asinterpolation or motion modeling.

Similarly, in other embodiments it is possible to infer the deformationusing other tracking sensors. In some embodiments, for example, it isknown in the art how to infer the deformation of an internal organ, suchas a beating heart or a breathing lung using, for example, an ultrasonicsensor. In other embodiments, it is possible to infer the deformationdue to the motion of the platform of the sensor using methodscollectively known in the art as simultaneous localization and mapping(SLAM).

Still referring to FIG. 6, most of the time, the deformation estimatedfrom the tracking sensor, is inexact, i.e., includes tracking errors andis of lower resolution than the resolution required for accurate imagereconstruction using the measurements of the measurement sensor and (4).Some embodiments of present disclosure correct the tracking error toproduce an accurate deformation and, as necessary, an accuratereconstructed image of the object.

FIG. 7 shows a schematic of reconstruction of a radar reflectivityimage, according to some embodiments of the present disclosure. In thisembodiment, the radar imaging system includes one or moreelectromagnetic sensors, such as radar arrays 710, and one or moreoptical sensors 720. The object 730, for example a human, moves anddeforms in front of the radar and the optical sensors, while the sensorsacquire snapshots. The data acquired by the optical sensor are processedby an optical tracking system 740, which produces a tracking of theobject and an inexact deformation 750 from snapshot to snapshot. Theoptical tracking system 740 may also map the optical deformation toobject prototypical pose for the object, i.e., determines the mapping F_(i) for each snapshot. This mapping is used together with the dataacquired in each radar snapshot to correct the inexact estimate of thedeformation and reconstruct 770 the radar reflectivity image of theobject 780. The reconstructed radar reflectivity image may berepresented in the prototypical pose by the system, by may be convertedand represented in any pose and with any modifications suitable to thesystem or its user, for example to highlight parts of the image forfurther examination 790, 792.

In such a manner, the radar imaging system includes an optical trackingsystem including the optical sensor to produce each deformation toinclude an optical transformation between points of an opticalreflectivity image including the object in the deformed shape and pointsof a prototypical optical reflectivity image including the object in thenominal shape. The processor of the radar imaging system determines thetransformation as a function of the optical transformation.

FIG. 8 shows an example of the motion and deformation of the object infront of the optical and radar sensors at each snapshot, according tosome embodiments of the present disclosure. In this example, a human 890walks in front of the sensors. The sensors obtain snapshots at differenttime instances, with the object in different pose. For example, in FIG.8, at each snapshot, the human is at a different position in front ofthe sensors, walking from left to right, and at a different pose,depending on the timing of each snapshot relative to the stride of thehuman.

FIG. 9 shows a schematic of the tracking performed by the optical sensorusing the example of FIG. 8, according to some embodiments of thepresent disclosure. Notably, there is a snapshot-based one-to-onecorrespondence between the deformation of the shape of the object in theoptical reflectivity image and corresponding transformation of the radarreflectivity image.

Each point on the human 900 is tracked by the camera at each timeinstant, and then mapped to the corresponding point in the prototypicalpose 990. Each point might or might not be visible in some snapshots.For example, points on the right shoulder 910, right knee 920, or rightankle 930 might always be visible, while points on the left hand 950might be occluded when the hand in behind the body and not visible tothe sensors 960. The tracking creates correspondences 980 between pointsin different snapshots and the corresponding point in the prototypicalimage. The correspondences are used to generate F _(i). If a point isnot visible to the sensor at some particular snapshot, e.g., 960, the F_(i) does not map this point to the radar grid, i.e., the correspondingcolumn of the operator contains all zeros. In that sense, F _(i)subsamples the prototypical radar image to only map the points that arevisible in the particular snapshot, as determined by the optical sensor.

Still referring to FIG. 9, some embodiments are based on recognitionthat an estimate F_(i) of F _(i) is inexact and contains errors. Theerrors can be modeled as F _(i)=P_(i)F_(i), where P_(i) is a correctionto the inexact estimate of the deformation. The correction P_(i) hassimilar structure as F_(i), but is less likely to deviate from theidentity. In other words, P_(i) is also a subsampled permutation or amore general operator which allows, e.g., blurring. However, becauseP_(i) models the errors in the motion tracking, and the motion trackingis in approximately correct, the mapping that P_(i) performs is onlyallowed to displace points a little bit away from the position theestimate F_(i) placed them in. In summary, F_(i), which is computed fromthe motion tracking part of the system, places the target grid inapproximately the right position, and P_(i) makes small corrections tothis placement. To estimate an accurate correction, P_(i), someembodiments of the present disclosure take the property that as P_(i)increasingly deviates from the identity, the less likely it is to beaccurate.

To that end, in some embodiments, the processor adjusts eachtransformation with a local error correction and determines concurrentlythe radar image of the object in the prototypical pose and each localerror correction. For example, the processor determines concurrently theradar image of the object in the prototypical pose and each local errorcorrection using one or combination of alternating minimization,projections, and constrained regularization.

Still referring to FIG. 9, those embodiments are based on recognitionthat the deformation error corrected using P_(i) is generally unknown.Otherwise, if the error is known, it would have been trivial to correctthe deformation error. Thus, the measurement system also estimates P_(i)from the snapshots, in order to correct the error. In other words, insome embodiments, the processor of the imaging system is configured tosolve

$\begin{matrix}{{\begin{bmatrix}y_{1} \\\vdots \\y_{T}\end{bmatrix} = {\begin{bmatrix}{A_{1}P_{1}F_{1}} \\\vdots \\{A_{T}P_{T}F_{T}}\end{bmatrix}x}},} & (5)\end{matrix}$

where all the P_(i) are unknown, in addition to x.

At least one key realization in the present disclosure is that eachunknown error correction P_(i) moves elements of F_(i)x, i.e., x asdeformed by the inexact deformation F_(i), to different locations in thesecond grid. Since the inexact deformation already has moved elements ofx to an approximately correct position, the deformation correction P_(i)should not move them too far from where F_(i) has located them. Thus,when estimating P_(i), solutions that cause large movement of theelements of F_(i)x should not be preferred.

Still referring to FIG. 9, on the other hand, the deformation correctionP_(i) should move elements of F_(i)x, such that they explain, i.e.,match, the measurement data y_(i). In order to explain the measurementdata, the corrected deformed signal P_(i)F_(i)x, when measured by theforward operator A_(i) should be as close as possible to the measureddata y_(i). Thus, when estimating P_(i), solutions that producemeasurements from the corrected deformed signal A_(i)F_(i)x that do notmatch the measured data y_(i) should not be preferred.

The preferences above represent different objectives that the desiredsolution should satisfy. Since these objectives are often competing,some embodiments of the present disclosure balance these objectives bydetermining a solution that combines them into a single cost function.To do so, some embodiments of the present disclosure determine a penaltyor cost function that increases the more the solution deviates from theobjective.

Still referring to FIG. 9, for example, to determine how well thesolution explains the measurements, some embodiments may use a norm ordistance function, computing the distance of the measurements of thecorrected deformed signal A_(i)P_(i)F_(i)x from the measured data y_(i).In some embodiments this norm may be a

₂ norm, typically denoted as ∥y_(i)−A_(i)P_(i)F_(i)x∥₂, although othernorms, such as a

₁ or

_(∞) norm, or distance or divergence functions, such as aKullbak-Leibler divergence, may be used. If, for a certain candidatesolution, the measurements of the corrected deformed signalA_(i)P_(i)F_(i)x do not match the measured data y_(i), then this norm,distance, or divergence will be large, thus penalizing that candidatesolution more than others. In contrast, if for a certain candidatesolution, the measurements of the corrected deformed signalA_(i)P_(i)F_(i)x match the measured data y_(i), then this norm,distance, or divergence will be small, not penalizing this solution.

Similarly, to determine if the solution causes large distortion in thecorrection of the elements of the signal F_(i)x, some embodiments use aregularization function R(P_(i)), which penalizes such solutions. Aregularization function is a term in the art describing functions thatdepend only on the solution—not the measured data—and have a large valuefor undesired solutions and a small value for desired solutions,similarly to how distance or divergence functions take a large or smallvalue depending on how well the solution matches the data, as describedabove.

Still referring to FIG. 9, some embodiments of the present disclosureuse regularization functions that take a large value if the deformationP_(i) moves elements of the deformed signal F_(i)x very far from theirposition in the imaging domain, wherein the imaging domain may have oneor two or more dimensions. For example, a regularization function mightinclude the sum of the distances that each of the elements moves withinthe image, wherein the distance may be a Euclidian (

₂) distance, or a Manhattan (

₁) distance, or a square-Euclidian distance (

₂ ²) or a maximum deviation (

_(∞)) distance or some other distance as appropriate for theapplication.

In order to balance the competing objective of matching the measurementsand determining deformations that do not move the elements too far fromtheir position, embodiments of the present disclosure try to minimize acost function that is the weighted sum of the two objectives

$\begin{matrix}{{\hat{P}}_{\iota},{x = {\underset{P_{i},x}{\arg\;\min}\left\{ {{\sum_{i}{{y_{i} - {A_{i}P_{i}F_{i}x}}}_{2}^{2}} + {\beta\;{R\left( P_{i} \right)}}} \right\}}},} & (6)\end{matrix}$

where the cost is added over all deformations in all snapshots, indexedby i, the weight β determines the balance between matching the data andregularization, and the minimization recovers both the deformationcorrections P_(i), and the signal x being imaged.

Still referring to FIG. 9, this minimization is non-convex, andtherefore difficult to solve. Furthermore, correct solutions shouldfurther enforce that P_(i) is a permutation, or a subsampledpermutation, such that the solution mathematically describes apermutation. However, determining permutations is a problem withcombinatorial complexity, which is difficult and very expensive tosolve.

In order to solve the problem, various embodiments of the presentdisclosure exploit a realization that, as corrections of the deformationare estimated, each correction of the deformation may produce anintermediate estimate of the deformed signal x_(i) that helps explainingthe measured data but does not exactly match the corrected deformedsignal P_(i)F_(i)x. Therefore, a separate cost component can be includedin the minimization (6) to balance how well the intermediate signalmatches the corrected deformed permutation:

$\begin{matrix}{{\hat{P}}_{\iota},{x = {\underset{P_{i},x_{i},x}{\arg\;\min}\left\{ {{\sum_{i}{{y_{i} - {A_{i}x_{i}}}}_{2}^{2}} + {\beta\;{R\left( P_{i} \right)}} + {\frac{\lambda}{2}{{x_{i} - {P_{i}F_{i}x}}}_{2}^{2}}} \right\}}},} & (7)\end{matrix}$

where the last term, ∥x_(i)−P_(i)F_(i)x∥₂ ², determines how well theintermediate signal x_(i) matches the corrected deformed permutationP_(i)F_(i)x. It should be noted that, while (7) uses the

₂ norm squared, i.e.,

₂ ², to quantify both how well the intermediate signal matches thecorrected deformed permutation and how well the measurements of theintermediate signal match the measurement data, other norms or distancescould be used, for example as enumerated above.

Still referring to FIG. 9, this realization is not obvious because itrelaxes the problem and introduces more unknown variables, theintermediate signals x_(i), i.e., makes the problem seemingly moredifficult to solve. However, the advantage of this relaxation is that itallows the use of optimal transport (OT) theory and algorithms to solvepart of the problem.

In particular, since P_(i) is a permutation, the last term in theminimization (7) can be expressed asΣ_(n,n′)(x_(i)[n]−(F_(i)x)[n′])²P_(i)[n, n′], where the notation u[n]selects the n^(th) element of a vector u, and the notation A [n, n′]selects the n^(th) row and n′^(th) column of P_(i). In this expression,n and n′, are indices on the first and second grid, respectively, i.e.,n′ indicates where the n^(th) element from the first grid will move toon the second grid. Furthermore, the regularization R(P_(i)) can beexpressed as Σ_(n,n′)∥l[n]−l′[n′]∥₂ ²P_(i)[n, n′], where l[n] and l′[n′]are the coordinates of points n and n′ in the first and the second grid,respectively.

Still referring to FIG. 9, a further realization is that P_(i)[n, n′]can be factored out of these two expressions, to combine them into asingle cost metric

C(x _(i) ,F _(i) x)[n,n′]=∥l[n]−l′[n′]∥₂ ²+(x _(i)[n]−(F _(i)x)[n′])²,  (8)

the product of which with P_(i)[n, n′] can be optimized over P_(i)[n,n′] being a permutation using OT algorithms known in the art. Using thisfactorization, the overall minimization (7) can be expressed as

$\begin{matrix}{{\hat{P}}_{\iota},x,{x_{i} = {\underset{x_{i},x}{\;\min}\left\{ {{\sum_{i}{{y_{i} - {A_{i}x_{i}}}}_{2}^{2}} + {\beta\;{\min\limits_{P_{i}}\left\langle {{C\left( {x_{i},{F_{i}x}} \right)},P_{i}} \right\rangle}}} \right\}}},} & (9)\end{matrix}$

where the notation

⋅,⋅

denotes the standard inner product, as well known in the art, namely thesum of the elementwise product of each component from the first argumentwith the corresponding component of the second argument, i.e.,

A, B

=Σ_(n,n′)A[n, n′]B[n, n′].

Still referring to FIG. 9, the OT literature provides algorithms andmethods to determine a permutation P_(i) that minimizes this innerproduct, known as the transport plan. The computation of this innerminimization in (9) is known in the art as the balanced OT problem

$\begin{matrix}{{{{OT}_{balanced}\left( {x,x_{i}} \right)} = {\min\limits_{P_{i}}\left\langle {{C\left( {x_{i},{F_{i}x}} \right)},P_{i}} \right\rangle}},} & (10)\end{matrix}$

in which the P_(i) that minimizes the OT problem is the OT plan. Solvingthe OT problem requires computing the optimal plan. The optimal planprovides a deformation in which all the elements of one snapshot aremapped to elements in the other snapshot. Thus, the OT problem does notallow for occlusion or otherwise missing elements, even though this isoften encountered in applications.

Other embodiments of the present disclosure may use an unbalanced OT ora partial OT problem in (9), to replace the balanced OT from (10), moregenerally

$\begin{matrix}{\overset{\hat{}}{P_{\iota}},x,{x_{i} = {\min\limits_{x_{i},x}\left\{ {{\sum_{i}{{y_{i} - {A_{i}x_{i}}}}_{2}^{2}} + {\beta\;{{OT}\left( {x,x_{i}} \right)}}} \right\}}},} & (11)\end{matrix}$

where OT(x, x_(i)) represents an OT problem which may include balanced,unbalanced, partial or some other OT problem known in the art. Thepartial or unbalanced OT literature provides algorithms and methods todetermine a subsampled P_(i), i.e., one in which certain parts of onesignal are occluded, i.e., are not part of the other signal and viceversa.

Still referring to FIG. 9, the OT problem is also known in the art asthe 2-D assignment problem because it only computes a transport planbetween a pair of signals, and it can be efficiently solved using alinear program. Solving (11) provides one approach to solving what isknown in the art as the N-D assignment problem, which simultaneouslycomputes all direct assignments between more than two signals, and whichis generally known to be very hard. At least one key realization thatprovides some embodiments of the present disclosure to solve the N-Dassignment problem is that instead of deforming all signals x_(i) tomatch all other signals, it is more efficient to only deform each signalx_(i) to match a common signal x, through the partially knowndeformation F_(i)x. This common signal serves in some sense as thetemplate from which all other signals are deformed.

By deforming each signal to only match a common signal, the solution nowonly requires computing deformations between pairs of signals—the commonone and each of the signals in the snapshots. Thus, the problem reducesto computing multiple pairwise assignments, i.e., 2-D assignments, sinceonly two signals are involved, instead of a single multi-signalassignment, i.e., N-D assignments. This is beneficial because 2-Dassignment problems are well-studied in the art and are much easier tosolve. A further realization is that this reduction works even if thedeformation is not known at all, and F_(i) is the identity, i.e.,implements no deformation.

Still referring to FIG. 9, the drawback in computing multiple 2-Dassignments can be that it increases the unknown variables of thesolution. A common signal x should now be computed in the process,making this reduction from N-D assignments to 2-D assignmentsnon-trivial. Some embodiments of the present disclosure rely on thefurther realization that the gradient of the 2-D assignment problem canbe computed in order to be able to use gradient descent methods tocompute x, thus making the reduction from N-D assignments to 2-Dassignments tractable.

The problem (11) involves minimizing over several variables, x, x_(i),P_(i), which are multiplicatively coupled. While the inner minimizationover P_(i) is understood in the art as the OT problem, the outerminimization over x, x_(i) is a non-convex problem that is difficult tosolve. In order to solve it, some embodiments of the present disclosurealternate between minimizing for x_(i), considering x fixed, andminimizing for x, considering x_(i) fixed. Other embodiments alternatebetween reducing the cost as a function of x_(i), considering x fixed,and reducing the cost as a function of x, considering x_(i) fixed.

FIG. 10A provides an overview schematic of the method to compute theminimization of the cost function in (11), according to some embodimentsof the present disclosure. An initial estimate of x and x_(i) is used asa starting point 1010. This estimate may be computed from themeasurements using well known methods in the art, including but notlimited to least squares inversion, matched filtering, back-projection,and sparse inversion, among others. In some embodiments, the initialestimates may be set to 0 or to a randomly generated signal.

FIG. 10B is an algorithm that can be used for reducing a cost functionwith respect to x_(i) 1025, according to some embodiments of the presentdisclosure.

FIG. 10C is an algorithm that can be used for reducing a cost functionwith respect to x 1020, according to some embodiments of the presentdisclosure.

Referring to FIG. 10A, FIG. 10B and FIG. 10C, these initial estimatesare updated by alternating between reducing the cost function withrespect to x 1020 and reducing the cost function with respect to x_(i)1025, until convergence 1070. An example embodiment of reducing the costfunction with respect to x_(i) 1025 is shown in FIG. 10B, and an exampleembodiment of reducing the cost function with respect to x 1020 is shownin FIG. 10C. An embodiment of the alternating update procedure issummarized in FIG. 10D. In these examples, the algorithms proceed for afixed number of iterations tMax. However, in other embodiments aconvergence criterion may be used instead, as described below.

Referring to FIG. 10A, some embodiments of the present disclosureconsider the system converged after a fixed number of iterations. Otherembodiments consider the change in the cost function after eachiteration and consider the system converged if the change is below acertain threshold for a fixed number of one or more iterations. Otherembodiments consider the gradient of the cost function after eachiteration and consider the system converged if the magnitude of thegradient is below a certain threshold for a fixed number of one or moreiterations. Other embodiments consider a combination of the above, orother conditions that may include total processing time, magnitude ofchange on the estimated signals and whether the computed transport planchanged from one iteration to the next.

Still referring to FIG. 10A, FIG. 10B and FIG. 10C, in order to reduceor minimize the cost as a function of either x 1020 or x_(i) 1025, someembodiments of the present disclosure compute the gradient of (11) withrespect to either x or x_(i), 1030 and 1030 respectively, and use thegradient to modify either x or x_(i), 1050 and 1055 respectively,towards reducing or minimizing the cost in (11). In some embodiments,the gradient is computed by evaluating an expression for the derivativethat has been analytically derived. Other embodiments may useauto-differentiation methods, now widely available in the art, that areable to compute a derivative of a function automatically, even if anexplicit analytic expression is not available. In some embodiments, thecomputation of the gradient requires the computation of the innerminimization, i.e., the OT problem and the use of the computed OT planin computing the derivative 1040 and 1045. In the example embodiments,the plan is computed in step 5 of FIG. 10B and FIG. 10C.

In order to compute the OT plan, some embodiments require thecomputation of an original and a target mass distribution for theproblem, as shown in steps 2 and 4 in FIG. 10B and steps 1 and 4 in FIG.10C. Some embodiments may, for example, use the signal estimates as amass distribution, or the points of the signal with values above acertain threshold, or a uniform distribution over the location in whichthe signal values are above a certain threshold, or a uniformdistribution over all possible signal locations, or a normalizeddistribution, or some other positive function over the signal value ateach location, or a combination thereof.

Still referring to FIG. 10A, FIG. 10B and FIG. 10C, in some embodiments,the general procedure to reduce or minimize the cost with respect toeither x or x_(i) requires using the current estimate of x and x_(i) tocompute the derivative at the current estimate, with respect to either xor x_(i), respectively, and then updating the estimate of either x orx_(i), respectively, according to this derivative, using a gradientstep, 1050 and 1055 respectively. This update step is

x ^(t+1) =x ^(t)−γ^(t)Σ_(i)∇_(x)ƒ(x ^(t) ,x _(i)),  (12)

x _(i) ^(t+1) =x _(i) ^(t)−γ^(t)∇_(x) _(i) ƒ(x,x _(i) ^(t)),  (13)

where ƒ(x, x_(i))=Σ_(i)∥y_(i)−A_(i)x_(i)∥₂ ²+βOT(x, x_(i)) is the costfunction in (11), ∇_(x) and ∇_(x) _(i) denote the gradient with respectto x and x_(i), respectively, γ^(t) is a gradient step size at step t,which may or may not be the same for (12) and (13), x^(t) and x_(i) ^(t)are the variables being updated at step t, x and x_(i) are the variablesconsidered fixed at the corresponding step, and x^(t+1) and x_(i) ^(t+1)are the updated variables.

Still referring to FIG. 10A, FIG. 10B and FIG. 10C, other embodimentsuse different methods to minimize (11), such as lifting to a higherspace, by estimating the outer product of x and x_(i) and imposinglow-rank structure on the resulting object. However, these approachesincrease the dimensionality of the problem significantly, and theresulting computational complexity, making them impractical for manyapplications.

After convergence, embodiments may produce in the output a combinationof the computed optimal transport plan, and the final estimate of x orx_(i) 1080.

FIG. 11A to FIG. 11E show experimentation performed on exampleembodiments, FIG. 11A shows a signal x in a prototypical position, FIG.11B shows a F_(i)x 1^(st) estimated deformation of the snapshot, FIG.11C shows a F_(i)x 2^(nd) estimated deformation of the snapshot, FIG.11D and FIG. E show actual deformations x_(i)=P_(i)F_(i)x of x observedby the acquisition system, according to some embodiments of the presentdisclosure. The objective in this experiment is to correct the estimatedapproximate deformation in FIGS. 11B and 11C in order to recover thesignal in FIG. 11A.

FIG. 12A shows the experimentation results of the experiment in FIG. 11Ato FIG. 11E, according to aspects of the present disclosure. The figureplots the reconstruction accuracy for different measurement rates, i.e.,number of measurements in the snapshots, compared to the size of thesignal. The reconstruction accuracy is reported with respect to thenormalized mean squared error in the recovered signal, where smallererror is better. The figure shows the comparison with a naïve approachin which the approximate deformation is assumed correct and a correctionis not computed while still attempting to recover the signal (labeled“Ignore Pi, noiseless” and demarcated with a solid black line and xmarkers). It also shows the comparison with an approach to correct theapproximate deformation known in the art, (labeled “Gradient, noiseless”and demarcated with a solid black line and +markers). Both comparisonsare assuming no measurement noise, an unrealistic assumption, favorableto these two methods.

The performance of embodiments of the present disclosure in the presenceof various levels of noise is demarcated using the dashed and lightercolored lines, label with “Input SNR=XXdB,” where XX denotes the inputnoise level. Since these are noisy experiments, the variability of themethods is demarcated using the shaded areas around the lines, whichrepresent one standard deviation above and below the average.

As evident in the figure, the prior art fails to accurately recover thesignal, even in ideal conditions, with noiseless measurements and highmeasurement rate. In contrast, embodiments of the present disclosure areable to reconstruct the signal with high fidelity assuming sufficientmeasurement rate given the noise level.

FIG. 12B shows further experimentation results demonstrating theperformance of the present embodiment under a fixed input SNR of 20 dBas the number of views increases. The figure plots the performance fortwo different measurement rates per view. As shown, the performanceimproves as the number of views—and, therefore, the total measurementrate—increases, and is better when the measurement rate per view ishigher. In each experiment plotted, the total measurement rate is equalto the number of views multiplied by the measurement rate per view.

FIG. 13 shows a hardware diagram of different components of the radarimaging system 1300, according to some embodiments of the presentdisclosure. The radar imaging system 1300 includes a processor 1320configured to execute stored instructions, as well as a memory 1340 thatstores instructions that are executable by the processor. The processor1320 can be a single core processor, a multi-core processor, a computingcluster, or any number of other configurations. The memory 1340 caninclude random access memory (RAM), read only memory (ROM), flashmemory, or any other suitable memory systems. The processor 1320 isconnected through a bus 1306 to one or more input and output devices.

These instructions implement a method for reconstructing radarreflectivity image of the object in the prototypical pose. To that end,the radar imaging system 1300 can also include a storage device 1330adapted to store different modules storing executable instructions forthe processor 1320. The storage device stores a deformation module 1331configured to estimate the deformation of the object in each snapshotusing measurements 1334 of the optical sensor data, a transformationmodule 1332 configured to obtain the transformations of the radarreflectivity images F_(i), which is an estimate of F _(i) the opticaldeformation; and reconstruction module 1333 configured to solve for x inEquation (5) above using the estimate F_(i) in place of the true F _(i),and optionally applying regularization, as described above. The storagedevice 1330 can be implemented using a hard drive, an optical drive, athumb drive, an array of drives, or any combinations thereof.

Still referring to FIG. 13, the radar imaging system 1300 includes aninput interface to receive measurements 1395 of the optical andelectromagnetic sensors. For example, in some implementations, the inputinterface includes a human machine interface 1310 within the radarimaging system 1300 that connects the processor 1320 to a keyboard 1311and pointing device 1312, wherein the pointing device 1312 can include amouse, trackball, touchpad, joy stick, pointing stick, stylus, ortouchscreen, among others.

Alternatively, the input interface can include a network interfacecontroller 1350 adapted to connect the radar imaging system 1300 throughthe bus 1306 to a network 1390. Through the network 1390, themeasurements 1395 can be downloaded and stored within the storage system1330 as training and/or operating data 1334 for storage and/or furtherprocessing.

Still referring to FIG. 13, the radar imaging system 1300 includes anoutput interface to render the prototypical radar reflectivity image ofthe object in the prototypical pose. For example, the radar imagingsystem 1300 can be linked through the bus 1306 to a display interface1360 adapted to connect the radar imaging system 1300 to a displaydevice 1365, wherein the display device 1365 can include a computermonitor, camera, television, projector, or mobile device, among others.

For example, the radar imaging system 1300 can be connected to a systeminterface 1370 adapted to connect the radar imaging system to adifferent system 1375 controlled based on the reconstructed radarreflectivity image. Additionally or alternatively, the radar imagingsystem 1300 can be connected to an application interface 1380 throughthe bus 1306 adapted to connect the radar imaging system 1300 to anapplication device 1385 that can operate based on results of imagereconstruction.

FIG. 14 is a schematic illustrating by non-limiting example a computingapparatus 1400 that can be used to implement some techniques of themethods and systems, according to embodiments of the present disclosure.The computing apparatus or device 1400 represents various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers.

The computing device 1400 can include a power source 1408, a processor1409, a memory 1410, a storage device 1411, all connected to a bus 1450.Further, a high-speed interface 1412, a low-speed interface 1413,high-speed expansion ports 1414 and low speed connection ports 1415, canbe connected to the bus 1450. Also, a low-speed expansion port 1416 isin connection with the bus 1450. Contemplated are various componentconfigurations that may be mounted on a common motherboard, bynon-limiting example, 1430, depending upon the specific application.Further still, an input interface 1417 can be connected via bus 1450 toan external receiver 1406 and an output interface 1418. A receiver 1419can be connected to an external transmitter 1407 and a transmitter 1420via the bus 1450. Also connected to the bus 1450 can be an externalmemory 1404, external sensors 1403, machine(s) 1402 and an environment1401. Further, one or more external input/output devices 1405 can beconnected to the bus 1450. A network interface controller (NIC) 1421 canbe adapted to connect through the bus 1450 to a network 1422, whereindata or other data, among other things, can be rendered on a third-partydisplay device, third-party imaging device, and/or third-party printingdevice outside of the computer device 1400.

Still referring to FIG. 14, contemplated is that the memory 1410 canstore instructions that are executable by the computer device 1400,historical data, and any data that can be utilized by the methods andsystems of the present disclosure. The memory 1410 can include randomaccess memory (RAM), read only memory (ROM), flash memory, or any othersuitable memory systems. The memory 1410 can be a volatile memory unitor units, and/or a non-volatile memory unit or units. The memory 1410may also be another form of computer-readable medium, such as a magneticor optical disk.

Still referring to FIG. 14, a storage device 1411 can be adapted tostore supplementary data and/or software modules used by the computerdevice 1400. For example, the storage device 1411 can store historicaldata and other related data as mentioned above regarding the presentdisclosure. Additionally, or alternatively, the storage device 1411 canstore historical data similar to data as mentioned above regarding thepresent disclosure. The storage device 1411 can include a hard drive, anoptical drive, a thumb-drive, an array of drives, or any combinationsthereof. Further, the storage device 1411 can contain acomputer-readable medium, such as a floppy disk device, a hard diskdevice, an optical disk device, or a tape device, a flash memory orother similar solid-state memory device, or an array of devices,including devices in a storage area network or other configurations.Instructions can be stored in an information carrier. The instructions,when executed by one or more processing devices (for example, processor1409), perform one or more methods, such as those described above.

The system can be linked through the bus 1450 optionally to a displayinterface or user Interface (HMI) 1423 adapted to connect the system toa display device 1425 and keyboard 1424, wherein the display device 1425can include a computer monitor, camera, television, projector, or mobiledevice, among others.

Still referring to FIG. 14, the computer device 1400 can include a userinput interface 1417 adapted to a printer interface (not shown) can alsobe connected through bus 1450 and adapted to connect to a printingdevice (not shown), wherein the printing device can include a liquidinkjet printer, solid ink printer, large-scale commercial printer,thermal printer, UV printer, or dye-sublimation printer, among others.

The high-speed interface 1412 manages bandwidth-intensive operations forthe computing device 1400, while the low-speed interface 1413 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In some implementations, the high-speed interface 1412 canbe coupled to the memory 1410, a user interface (HMI) 1423, and to akeyboard 1424 and display 1425 (e.g., through a graphics processor oraccelerator), and to the high-speed expansion ports 1414, which mayaccept various expansion cards (not shown) via bus 1450. In theimplementation, the low-speed interface 1413 is coupled to the storagedevice 1411 and the low-speed expansion port 1415, via bus 1450. Thelow-speed expansion port 1415, which may include various communicationports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupledto one or more input/output devices 1405, and other devices a keyboard1424, a pointing device (not shown), a scanner (not shown), or anetworking device such as a switch or router, e.g., through a networkadapter.

Still referring to FIG. 14, the computing device 1400 may be implementedin a number of different forms, as shown in the figure. For example, itmay be implemented as a standard server 1426, or multiple times in agroup of such servers. In addition, it may be implemented in a personalcomputer such as a laptop computer 1427. It may also be implemented aspart of a rack server system 1428. Alternatively, components from thecomputing device 1400 may be combined with other components such as theembodiment of FIG. 13. Each of such devices may contain one or more ofthe computing device 1300 and the device 1400, and an entire system maybe made up of multiple computing devices communicating with each other.

Features

An aspect can include the measurement sensor captures measurements ofthe object deforming in the scene over the multiple time steps for thetime period, by continuously capturing snapshots of the object for themultiple steps for the period of time, and sequentially transmits themeasurement data to the processor, where at each time step, the objectexhibits a different deformation for the multiple time steps. Wherein anaspect is the tracking system tracks the deformable object during thesame time period or a different time period, as that of the measurementsensor capturing snapshots of the object deforming.

Another aspect the deformation is wholly or partly caused by the objectmoving in the scene or that the deformation is wholly or partly causedby the measurement sensor moving while capturing the scene. Anotheraspect the system is a coherent imaging system, such as a radar imagingsystem, a magnetic resonance imaging system or an ultrasound imagingsystem. Further, an aspect is the correction to the estimates of thedeformation of the object for each time step is computed using anoptimization that minimizes a cost function that includes an amount of adistance of how far the estimated deformation moves elements of theobject, and a level of a measurement of how the deformed object matchesto the measurements of the tracking system. Wherein a further aspect isthe matching the measurements of the corrected deformation of the objectfor each time step to measurements in the acquired snapshot of theobject for that time step is based on using a cost function thatpenalizes an amount of a distance between measurements of the correcteddeformations of the object and measurements in the acquired snapshot ofthe object for that time step. Wherein another further aspect is theestimating of the corrected deformation over other correcteddeformations for that time step, is according to the distance betweenthe corrected deformation and the initial estimate of the deformation,and based on using a cost function that penalizes more the correctionsto the deformations, in which elements of the object move an amount of adistance farther, when compared to their deformed location.

An aspect is that an optimal transport problem, which includes a costthat penalizes deformations according to an amount of a distance of howfar these deformations move elements of the object image from theirposition and a cost that penalizes deformations according to a level ofa matching score of how well the measurements of the correcteddeformations of the object match to the measurements of the trackingsystem. The aspect is that the object deforming in the scene is one of,a mammal including a human, an amphibian, a bird, a fish, aninvertebrate or a reptile, wherein the object deforming in the scene isan organ inside a body of the human, an organ inside of the amphibian,an organ inside of the bird, an organ inside of the fish, an organinside of the invertebrate or an organ inside of the reptile.

Another aspect is the final estimate of the deformation of thedeformable object, the final image of the object, or both, are labeledas an object report, and outputted to, and received by, a communicationnetwork associated with an entity such as an operator of the system, theoperator generates at least one action command that is sent to, andreceived by a controller associated with the system which implements thegenerated at least one action command, resulting in changing a propertyof the object based upon the object report. Wherein an aspect is theproperty of the object includes one or a combination of, a defect in theobject, a medical condition of the object, a presence of a weapon on theobject or a presence of an undesirable artifact on the object. Whereinanother aspect is the at least one action command includes one or acombination of, a level of an object defect inspection from a set ofdifferent levels of object defect inspections, a level of an objectmedical testing from a set of different levels of object medicaltesting, a level of an object security and safety inspection from a setof different levels of object security and safety inspections.

Another aspect is that the tracking sensor has one or combination of anoptical camera, a depth camera and an infrared camera, and wherein theelectromagnetic sensor includes one or combination of a mmWave radar, aThz imaging sensor, and a backscatter X-ray sensor, and wherein. Stillanother aspect is that the electromagnetic sensor is a plurality ofelectromagnetic sensors having a fixed aperture size, wherein theprocessor estimates the radar image of the object for each time step ofthe multiple time steps from the radar reflectivity image of the sceneby combining measurements of each electromagnetic sensor from theplurality of electromagnetic sensors. Wherein the plurality ofelectromagnetic sensors are moving according to known motions, andwherein the processor adjusts the transformation of the radarreflectivity image of the object acquired by the plurality ofelectromagnetic sensors at the corresponding time step based on theknown motions of the plurality of electromagnetic sensors for thecorresponding time step. Wherein an aspect is a resolution of the radarreflectivity image of the scene is greater than resolutions of theinitial estimates of the deformation of the object in each time step.

Definitions

Types of Radar and radar sensors: Radar can come in a variety ofconfigurations in an emitter, a receiver, an antenna, wavelength, scanstrategies, etc. For example, some radar can include Bistatic radar,Continuous-wave radar, Doppler radar, Frequency Modulated ContinuousWave (Fm-cw) radar, Monopulse radar, Passive radar, Planar array radar,pulse radars with arbitrary waveforms, Pulse-doppler, multistaticradars, Synthetic aperture radar, Synthetically thinned aperture radar,Over-the-horizon radar with Chirp transmitter, interferometric radars,polarimetric radars, array-based radars or MIMO (Multiple Input MultipleOutput) radars (MIMO), etc. Contemplated is incorporating one or moretypes of radar and radar sensors with one or more embodiments of theradar imaging system of the present disclosure.

Embodiments

The following description provides exemplary embodiments only, and isnot intended to limit the scope, applicability, or configuration of thedisclosure. Rather, the following description of the exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing one or more exemplary embodiments.Contemplated are various changes that may be made in the function andarrangement of elements without departing from the spirit and scope ofthe subject matter disclosed as set forth in the appended claims.

Although the present disclosure has been described with reference tocertain preferred embodiments, it is to be understood that various otheradaptations and modifications can be made within the spirit and scope ofthe present disclosure. Therefore, it is the aspect of the append claimsto cover all such variations and modifications as come within the truespirit and scope of the present disclosure.

1. An imaging system comprising: a tracking system to track a deformingobject within a scene over multiple time steps for a period of time toproduce an initial estimate of a deformation of the object for each timestep; a measurement sensor captures measurement data by capturingsnapshots of the object deforming in the scene over the multiple timesteps for the time period; and a processor that calculates, for themeasurement data, deformation information of the deforming object, basedon using each acquired snapshot of the object having measurements of theobject in a deformation for that time step, to produce a set ofmeasurements of the object with deformed shapes over the multiple timesteps, and that for each time step of the multiple time steps, theprocessor sequentially calculates deformation information of object, bycomputing a correction to the estimates of the deformation of theobject, wherein the correction includes matching measurements of thecorrected deformation of the object for each time step to measurementsin the acquired snapshot of the object for that time step, and for eachtime step, select a corrected deformation over other correcteddeformations for that time step, according to a distance between thecorrected deformation and the initial estimate of the deformation, toobtain a final estimate of the deformation of the deformable objectmoving in the scene and a final image of the object moving within thescene.
 2. The imaging system according to claim 1, wherein themeasurement sensor captures measurements of the object deforming in thescene over the multiple time steps for the time period, by continuouslycapturing snapshots of the object for the multiple steps for the periodof time, and sequentially transmits the measurement data to theprocessor, where at each time step, the object exhibits a differentdeformation for the multiple time steps.
 3. The imaging system accordingto claim 1, wherein the tracking system tracks the deformable objectduring the same time period or a different time period, as that of themeasurement sensor capturing snapshots of the object deforming.
 4. Theimaging system according to claim 1 wherein the deformation is wholly orpartly caused by the object moving in the scene, or wherein thedeformation is wholly or partly caused by the measurement sensor movingwhile capturing the scene.
 5. The imaging system of claim 1, wherein thesystem is a coherent imaging system, such as a radar imaging system, amagnetic resonance imaging system or an ultrasound imaging system. 6.The imaging system of claim 1, wherein the tracking system includes atleast one tracking sensor that is one or combination of an opticalcamera, a depth camera and an infrared camera, and wherein themeasurement sensor is at least one electromagnetic sensor that includesone or combination of a mmWave radar, a Thz imaging sensor, and abackscatter X-ray sensor.
 7. The imaging system of claim 1, wherein thecorrection to the estimates of the deformation of the object for eachtime step is computed using an optimization that minimizes a costfunction that includes an amount of a distance of how far the estimateddeformation moves elements of the object, and a level of a measurementof how the deformed object matches to the measurements of the trackingsystem.
 8. The imaging system of claim 7, wherein the matching themeasurements of the corrected deformation of the object for each timestep to measurements in the acquired snapshot of the object for thattime step is based on using a cost function that penalizes an amount ofa distance between measurements of the corrected deformations of theobject and measurements in the acquired snapshot of the object for thattime step.
 9. The imaging system of claim 7, wherein the estimating ofthe corrected deformation over other corrected deformations for thattime step, is according to the distance between the correcteddeformation and the initial estimate of the deformation, and based onusing a cost function that penalizes more the corrections to thedeformations, in which elements of the object move an amount of adistance farther, when compared to their deformed location.
 10. Theimaging system of claim 1, further comprising: an optimal transportproblem, which includes a cost that penalizes deformations according toan amount of a distance of how far these deformations move elements ofthe object image from their position, and a cost that penalizesdeformations according to a level of a matching score of how well themeasurements of the corrected deformations of the object match to themeasurements of the tracking system.
 11. The imaging system of claim 1,wherein the object deforming in the scene is one of, a mammal includinga human, an amphibian, a bird, a fish, an invertebrate or a reptile,wherein the object deforming in the scene is an organ inside a body ofthe human, an organ inside of the amphibian, an organ inside of thebird, an organ inside of the fish, an organ inside of the invertebrateor an organ inside of the reptile.
 12. The imaging system of claim 1,wherein the final estimate of the deformation of the deformable object,the final image of the object, or both, are labeled as an object report,and outputted to, and received by, a communication network associatedwith an entity such as an operator of the system, the operator generatesat least one action command that is sent to, and received by acontroller associated with the system which implements the generated atleast one action command, resulting in changing a property of the objectbased upon the object report.
 13. The imaging system of claim 12,wherein the property of the object includes one or a combination of, adefect in the object, a medical condition of the object, a presence of aweapon on the object or a presence of an undesirable artifact on theobject.
 14. The imaging system of claim 12, wherein the at least oneaction command includes one or a combination of, a level of an objectdefect inspection from a set of different levels of object defectinspections, a level of an object medical testing from a set ofdifferent levels of object medical testing, a level of an objectsecurity and safety inspection from a set of different levels of objectsecurity and safety inspections.
 15. An image processing method,comprising: tracking a deforming object within the scene over multipletime steps for a period of time via a tracking system to estimate adeformation of the object for each time step; acquiring measurement databy continuously capturing snapshots of the object deforming in the sceneover the multiple time steps for the period of time; calculating, forthe measurement data, deformation information of the deforming object,such that each acquired snapshot of the object includes measurements ofthe object in a deformation for that time step, to produce a set ofmeasurements of the object with deformed shapes over the multiple timesteps; calculating deformation information of the object, by computing acorrection to the estimates of the deformation of the object for eachtime step for the multiple time steps, such that the correction includesmatching measurements of the corrected deformation of the object foreach time step to measurements in the acquired snapshot of the objectfor that time step, and for each time step, select a correcteddeformation over other corrected deformations for that time step,according to a distance between the corrected deformation and theinitial estimate of the deformation, to obtain a final estimate of thedeformation of the deformable object moving in the scene and a finalimage of the object moving within the scene, which are stored.
 16. Aproduction apparatus comprising: a tracking system to track a deformingobject within a scene over multiple time steps for a period of time toproduce an initial estimate of a deformation of the object for each timestep; a measurement sensor captures measurement data by capturingsnapshots of the object deforming in the scene over the multiple timesteps for the time period; and a processor that calculates, for themeasurement data, deformation information of the deforming object, basedon using each acquired snapshot of the object having measurements of theobject in a deformation for that time step, to produce a set ofmeasurements of the object with deformed shapes over the multiple timesteps, and for each time step, the processor sequentially calculatesdeformation information of the object, by computing a correction to theestimates of the deformation of the object for each time step for themultiple time steps, such that the correction includes matchmeasurements of the corrected deformation of the object for each timestep to measurements in the acquired snapshot of the object for thattime step, and for each time step, select a corrected deformation overother corrected deformations for that time step, according to a distancebetween the corrected deformation and the initial estimate of thedeformation, to obtain a final estimate of the deformation of thedeformable object moving in the scene and a final image of the objectmoving within the scene, which are stored.
 17. A radar system toestimate a deformation of a deformable object moving in a scene,comprising: a tracking system having a tracking sensor to track thedeforming object over multiple time steps for a period of time toproduce an initial estimate of the deformation of the object for eachtime step of the multiple time steps, such that at each time stepincludes a different deformation; an electromagnetic sensor thatcaptures measurements of the object deforming in the scene over themultiple time steps for the time period as measurement data, bycapturing snapshots of the object moving over the multiple time steps aprocessor that calculates, for the measurement data, deformationinformation of the deforming object, wherein the electromagnetic sensorcaptures snapshots of the object deforming over the multiple time steps,each acquired snapshot of the object in the measurement data includesmeasurements of the object in a deformation for that time step, toproduce a set of measurements of the object with deformed shapes overthe multiple time steps, and wherein for each time step for the multipletime steps, the processor sequentially calculates deformationinformation of object, by computing a correction to the estimates of thedeformation of the object for each time step for the multiple timesteps, such that the correction includes matching measurements of thecorrected deformation of the object for each time step to measurementsin the acquired snapshot of the object for that time step, and for eachtime step, select a corrected deformation over other correcteddeformations for that time step, according to a distance between thecorrected deformation and the initial estimate of the deformation, toobtain a final estimate of the deformation of the deformable objectmoving in the scene and a final image of the object moving within thescene; and output the final estimate of the deformation of thedeformable object to one or more components of at least one output ofthe radar system or to another system associated with the radar system.18. The radar system of claim 17, wherein the electromagnetic sensor isa plurality of electromagnetic sensors having a fixed aperture size,wherein the processor estimates the radar image of the object for eachtime step of the multiple time steps from the radar reflectivity imageof the scene by combining measurements of each electromagnetic sensorfrom the plurality of electromagnetic sensors.
 19. The radar system ofclaim 17, wherein the plurality of electromagnetic sensors are movingaccording to known motions, and wherein the processor adjusts thetransformation of the radar reflectivity image of the object acquired bythe plurality of electromagnetic sensors at the corresponding time stepbased on the known motions of the plurality of electromagnetic sensorsfor the corresponding time step.
 20. The radar system of claim 17,wherein a resolution of the radar reflectivity image of the scene isgreater than resolutions of the initial estimates of the deformation ofthe object in each time step.
 21. A radar imaging method to reconstructa radar reflectivity image of a scene including an object deformingwithin the scene, having steps of tracking the deforming object overmultiple time steps for a period of time using a tracking system toproduce an initial estimate of a deformation of the object for each timestep, where at each time step there is a different deformation, and thestep of acquiring measurement data by continuously capturing snapshotsof the object deforming in the scene over the multiple time steps forthe period of time, and another step of calculating, for the measurementdata, deformation information of the deforming object, such that eachacquired snapshot of the object includes measurements of the object in adeformation for that time step, to produce a set of measurements of theobject with deformed shapes over the multiple time steps, the methodcomprising: calculating deformation information of the object, bycomputing a correction to the estimates of the deformation of the objectfor each time step for the multiple time steps, such that the correctionincludes matching measurements of the corrected deformation of theobject for each time step to measurements in the acquired snapshot ofthe object for that time step, and for each time step, select acorrected deformation over other corrected deformations for that timestep, according to a distance between the corrected deformation and theinitial estimate of the deformation, which is based on using anoptimization that minimizes a cost function that includes an amount of adistance of how far the estimated deformation moves elements of theobject, and a level of a measurement of how the deformed object matchesto the measurements of the tracking system, to obtain a final estimateof the deformation of the deformable object moving in the scene and afinal image of the object moving within the scene; and outputting thefinal estimate of the deformation of the deformable object and the finalradar image of the object, to the radar system or another systemassociated with the radar system.